Enhanced vision system and method

ABSTRACT

In one implementation, an enhanced vision system includes a portable user device, a base station including a hardware processor and a memory storing a virtual effects rendering software code, and a display device communicatively coupled to the base station. The hardware processor executes the virtual effects rendering software code to detect the presence of the portable user device in a real-world environment, obtain a mapping of the real-world environment, and identify one or more virtual effect(s) for display in the real-world environment. The hardware processor further executes the virtual effects rendering software code to detect actuation of the portable user device, and to control the display device to display the virtual effect(s) in the real-world environment based on the mapping, and the position and orientation of the portable user device during the detected actuation.

BACKGROUND

In augmented reality (AR), the appearance of real-world objects and/orenvironments can be digitally modified using virtual imagery to providea user with the illusion of having enhanced vision, such as the illusionof having “x-ray” vision enabling the user to see features obscured byan opaque surface. AR is increasingly used to produce entertainmentexperiences that are more immersive and engaging. Moreover, AR can beused to modify images of the real-world through augmentation in waysthat have a wide variety of practical applications beyond entertainment.However, a user wishing to experience the enhanced vision made possibleby AR must typically view real-world objects through AR glasses or usingan AR headset in order to see those real-world objects overlaid byvirtual projections.

Unfortunately, AR glasses and headsets can be costly and inconvenient towear. In addition, the increased concern over the spread of communicabledisease will likely mandate burdensome sanitation procedures in usageenvironments in which wearable AR viewing equipment is shared bymultiple users. Moreover, requiring the use of an AR headset or glassesto enjoy an enhanced vision experience effectively precludes multipleusers from sharing the same experience. Consequently, there is a need inthe art for an AR solution enabling one or more users to enjoy enhancedvision without requiring the user or users to don AR eyewear orheadgear.

SUMMARY

There are provided enhanced vision systems and methods, substantially asshown in and/or described in connection with at least one of thefigures, and as set forth more completely in the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A shows a diagram of an exemplary enhanced vision system,according to one implementation;

FIG. 1B shows a diagram of an exemplary enhanced vision system,according to another implementation;

FIG. 2 shows a diagram including a more detailed exemplaryrepresentation of a portable user device of the enhanced vision systemsshown in FIGS. 1A and 1B, according to one implementation;

FIG. 3 is a flowchart presenting an exemplary enhanced vision method,according to one implementation;

FIG. 4A shows an exemplary utility application for an enhanced visionsystem, according to one implementation; and

FIG. 4B shows an exemplary game application for an enhanced visionsystem, according to one implementation.

DETAILED DESCRIPTION

The following description contains specific information pertaining toimplementations in the present disclosure. One skilled in the art willrecognize that the present disclosure may be implemented in a mannerdifferent from that specifically discussed herein. The drawings in thepresent application and their accompanying detailed description aredirected to merely exemplary implementations. Unless noted otherwise,like or corresponding elements among the figures may be indicated bylike or corresponding reference numerals. Moreover, the drawings andillustrations in the present application are generally not to scale, andare not intended to correspond to actual relative dimensions.

The present application discloses enhanced vision systems and methodsthat overcome the drawbacks and deficiencies in the conventional art.According to the present novel and inventive concepts, a system user canexperience the enhanced vision made possible by augmented reality (AR)without the discomfort or inconvenience associated with wearing AReyewear or headgear. Instead, the user may actuate a portable device,such as a handheld device simulating a laser pointer or flashlight, forexample, to point at an object or a region of the user's real-worldenvironment. Based on the particular application of the enhanced visionsystem, a mapping of the environment in which the portable user deviceis utilized, the location and orientation of the portable user devicewhen it is actuated, and in some implementations the perspective of theuser with respect to the object or region pointed to, virtual effectsproviding an enhanced vision experience am displayed to the user. Theenhanced vision solution disclosed by the present application mayadvantageously be utilized in a wide variety of use cases including, butnot limited to, AR games, architectural inspection, and industrial andresidential safety.

It is noted that, as used herein, the feature “virtual effect” refers toone or more virtual images used to overlay an image of a real-worldobject. Moreover, “virtual effect” refers to one or more virtual imagesin the form of environmental features, such as lighting, color, orstructural/architectural features of a venue, or to simulations ofpersons, avatars, characters, caricatures of a person, animals, plants,and living things of various species or varieties, as well as inanimateobjects.

FIG. 1A shows a diagram of exemplary enhanced vision system 100A,according to one implementation. As shown in FIG. 1A, enhanced visionsystem 100A includes base station 102 having hardware processor 104 andmemory 106 implemented as a non-transitory storage device. In addition,in some implementations, base station 102 may include one or both ofmapping device 108 and transceiver 116. According to the implementationshown in FIG. 1A, virtual effects rendering software code 120 andenhanced vision database 110 are stored in memory 106. Enhanced visiondatabase 110 includes virtual effects 112 and real-world environmentalmaps 114.

In addition to base station 102, enhanced vision system 100A alsoincludes portable user device 130, and display device 126Acommunicatively coupled to base station 102. Furthermore, in someimplementations, enhanced vision system 100A may include tracking device122. It is noted, that as defined for the purposes of the presentapplication, the expression “communicatively coupled” may meanphysically integrated with, or physically discrete from but incommunication with. Thus, display device 126A may be integrated withbase station 102, or may be adjacent to or remote from base station 102while being in wired or wireless communication with base station 102.

As further shown in FIG. 1A, enhanced vision system 100A is implementedwithin real-world environment 140 including one or more objects orstructures, represented by wall structure 142 in FIG. 1A. Portable userdevice 130 is configured to be used by user 144. Also shown in FIG. 1Aare tracking data 124 received by base station 102 from optionaltracking device 122, display data 128 provided as an output to displaydevice 126A by base station 102, and optional wireless communicationlink 146 communicatively coupling base station 102 and portable userdevice 130.

According to the exemplary implementation shown in FIG. 1A, portableuser device 130 may be a handheld device, such as a replica of a laserpointer or flashlight, for example. It is noted that the term “replica”refers to an object that physically resembles another object having aspecific functionality, while lacking that functionality. Thus, areplica of a laser pointer or flashlight would appear to a casualobserver to appear to be that object despite not being capable ofemitting light. Analogously, a replica of a weapon such as a light sabermight be a toy or prop resembling a weapon but not designed to be usedin actual combat.

However, in other implementations, portable user device 130 may be awearable device, such as a replica of a scanner, camera, or other typeof sensing device worn, for example, on a lanyard, by user 144. Basestation 102 is configured to track portable user device 130 inreal-world environment 140. In some implementations, portable userdevice 130 may be a smart device, such as a smartphone, tablet computer,or any of the exemplary replica devices identified above that includesmart technology similar to that included in a smartphone or tabletcomputer. In some of those implementations, base station 102 may trackportable user device 130 by receiving position and orientation data fromportable user device 130 via wireless communication link 146, where theposition and orientation data describe the location, yaw, pitch, androll of portable user device 130.

In some implementations, portable user device 130 may be a replicadevice lacking smart technology. In those implementations, enhancedvision system 100A may include tracking device 122 communicativelycoupled to base station 102 and configured to determine the position andorientation of portable user device 130 in real-world environment 140.For example, portable user device 130 may include one or more infrared(IR) emitters tracked by tracking device 122 in the form of an IR cameraor other IR sensor. Alternatively, or in addition, tracking device 122may include a camera, camera array, or one or more other type of opticalsensor for determining the position and orientation of portable userdevice 130 in real-world environment 140.

As another alternative, or in addition, tracking device 122 may includemultiple components distributed within real-world environment 140 andconfigured to perform radio-signal triangulation to determine theposition and orientation of portable user device 130. As yet anotheralternative, base station 102 may utilize optional mapping device 108configured to perform simultaneous localization and mapping (SLAM) todetermine the position and orientation of portable user device 130 inreal-world environment 140.

In addition to determining the position and orientation of portable userdevice 130 in real-world environment 140, enhanced vision system 100Amay further determine the viewing perspective of user 144 of portableuser device 130. For example, in some implementations, tracking device122 may be configured to perform eye tracking or skeleton tracking ofuser 144 in real-world environment 140. Alternatively, or in addition,tracking data 124 generated by tracking device 122 may include opticaldata enabling virtual effects rendering software code 120, executed byhardware processor 104 of base station 102, to estimate the viewingperspective of user 144 based on the distance separating the eyes ofuser 144 from portable user device 130, and/or based on the headposition of user 144.

Base station 102 is configured to communicate with tracking device 122or directly with portable user device 130 to detect actuation ofportable user device 130 by user 144. For example, where portable userdevice 130 is a replica of a flashlight, actuation of portable userdevice 130 may include pointing portable user device 130 at an object orstructure within real-world environment 140, such as wall structure 142for example, and pressing a button or other actuation controller onportable user device 130 to mimic illuminating the replica flashlight.Analogously, where portable user device 130 is a replica of a laserpointer, or other pointing device, actuation of portable user device 130may include pointing portable user device 130 at an object or structurewithin real-world environment 140, such as wall structure 142 forexample, and pressing a button or other controller on portable userdevice 130 to mimic actuation of the replica pointing device.

When actuation of portable user device 130 is detected, base station 102controls display device 126A, using display data 128, to display one ormore virtual effects 112 in real-world environment 140 based on amapping of real-world environment 140, and at least the position andorientation of portable user device 130 during the detected actuation.Moreover, in some implementations, virtual effects 112 may be displayedunder the control of base station 102 using the viewing perspective ofuser 144, for example by utilizing an image-warping technique, as knownin the art, to correct for the viewing perspective of user 144.

In some implementations, as shown in FIG. 1A, display device 126A mayinclude one or more projectors, and controlling display device 126Aprojects one or more virtual effects 112 onto the surface of wallstructure 142 or another surface within real-world environment 140. Whenimplemented as one or more projectors, display device 126A may include astand-alone wide-field projection system, such as a spinningpoint-of-view (POV) projected illumination “spinning scan line,” forexample. Alternatively, display device 126A may be implemented as afisheye lens projector, or as multiple stitched projection-mapped videoprojectors.

It is noted that, although the present application refers to virtualeffects rendering software code 120 as being stored in memory 106 forconceptual clarity, more generally, memory 106 may take the form of anycomputer-readable non-transitory storage medium. The expression“computer-readable non-transitory storage medium,” as used in thepresent application, refers to any medium, excluding a carrier wave orother transitory signal that provides instructions to hardware processor104 of base station 102. Thus, a computer-readable non-transitory mediummay correspond to various types of media, such as volatile media andnon-volatile media, for example. Volatile media may include dynamicmemory, such as dynamic random access memory (dynamic RAM), whilenon-volatile memory may include optical, magnetic, or electrostaticstorage devices. Common forms of computer-readable non-transitory mediainclude, for example, optical discs, RAM, programmable read-only memory(PROM), erasable PROM (EPROM), and FLASH memory.

In some implementations, real-world environment 140 may take the form ofan indoor venue. Such indoor venues may include a personal residence, afactory or other industrial facility, or a film or broadcast studio, toname a few examples. It is noted that although FIG. 1A explicitly showsreal-world environment 140 to include only wall structure 142, thatsimplified representation is provided merely for conceptual clarity.More generally, real-world environment 140 may include multiplestructures, such as walls corresponding to wall structure 142, as wellas a ceiling, floor, and one or more objects, such as articles offurniture, art or decorative objects, and manufacturing equipment orother machinery, to name a few examples.

FIG. 1B shows a diagram of exemplary enhanced vision system 100B,according to another implementation. It is noted that enhanced visionsystem 100B, in FIG. 1B, corresponds in general to enhanced visionsystem 100A, in FIG. 1A, and may share any of the characteristicsattributed to that corresponding system by the present disclosure. It isfurther noted that any feature in FIG. 1B identified by a referencenumber identical to a reference number appearing in FIG. 1A correspondsto that previously described feature and may share any of thecharacteristics attributed to it above.

According to the exemplary implementation shown in FIG. 1B, and incontrast to the implementation shown in FIG. 1A, display device 126B ofenhanced vision system 100B is wall structure 142 of real-wordenvironment 140, where wall structure 142 includes multiple displayelements 148. In the exemplary implementation shown in FIG. 1B, basestation 102 controls display device 126B using display data 128 toactivate at least some of display elements 148 of wall structure 142, torender one or more virtual effects 112. In various implementations, wallstructure 142 may be a light-emitting diode (LED) wall including displayelements 148 in the form of LEDs, or may be an organic light-emittingdiode (OLED) wall including display elements 148 in the form of OLEDs.

FIG. 2 shows a more detailed representation of exemplary portable userdevice 230 in combination with base station 202, according to oneimplementation. As shown in FIG. 2 , portable user device 230 iscommunicatively coupled to base station 202 by wireless communicationlink 246. Base station 202 includes hardware processor 204 and memory206 implemented as a non-transitory storage device. In addition, in someimplementations, base station 202 may include one or both of mappingdevice 208 and transceiver 216 a. As further shown in FIG. 2 , memory206 contains virtual effects rendering software code 220 a and enhancedvision database 210 storing virtual effects 212 and real-worldenvironmental maps 214.

Portable user device 230 includes hardware processor 234 and memory 236implemented as a non-transitory storage device storing virtual effectsrendering software code 220 b. As also shown in FIG. 2 , portable userdevice 230 may include any or all of transceiver 216 b, one or morecameras 250 (hereinafter “camera(s) 250”), radio-frequencyidentification (RFID) reader 252, one or more position/location sensors238 (hereinafter “P/L sensor(s) 238”), and display 232 receiving displaydata 228 from virtual effects rendering software code 220 b. Also shownin FIG. 2 is tracking data 224 generated by portable user system 230 andtransmitted to base station 202 via wireless communication link 246.

Base station 202 having hardware processor 204, memory 206 includingenhanced vision database 210 storing virtual effects 212 and real-worldenvironmental maps 214, and optional mapping device 208, corresponds ingeneral to base station 102 having hardware processor 104, memory 106including enhanced vision database 110 storing virtual effects 112 andreal-world environmental maps 114, and optional mapping device 108, inFIGS. 1A and 1B. Thus, base station 202, hardware processor 204, memory206, enhanced vision database 210, virtual effects 212, real-worldenvironmental maps 214, and optional mapping device 208 may share any ofthe characteristics attributed to respective base station 102, hardwareprocessor 104, memory 106 enhanced vision database 110, virtual effects112, real-world environmental maps 114, and optional mapping device 108by the present disclosure, and vice versa.

In addition, transceiver 216 a and virtual effects rendering softwarecode 220 a of base station 202 correspond respectively in general totransceiver 116 and virtual effects rendering software code 120, inFIGS. 1A and 1B. Consequently, transceiver 116 and virtual effectsrendering software code 120 may share any of the characteristicsattributed to respective transceiver 216 a and virtual effects renderingsoftware code 220 a by the present disclosure, and vice versa. It isalso noted that tracking data 224, display data 228, and wirelesscommunication link 246, in FIG. 2 , correspond respectively in generalto tracking data 124, display data 128, and wireless communication link146, in FIGS. 1A and 1B, and those corresponding features may share anyof the characteristics attributed to either corresponding featureherein.

Portable user device 230 corresponds in general to portable user device130, in FIGS. 1A and 1B, and those corresponding features may share anyof the characteristics attributed to either corresponding feature by thepresent disclosure. Thus, like portable user device 230, portable userdevice 130 may include features corresponding to hardware processor 234,transceiver 216 b, camera(s) 250, RFID reader 252, P/L sensor(s) 238,display 232, and memory 236 storing virtual effects rendering softwarecode 220 b. Moreover, like portable user device 130, portable userdevice 230 may take a variety of forms. For example, as described aboveby reference to FIG. 1A, portable user device 130/230 may be a handhelddevice, such as a replica of a laser pointer or flashlight, for example,or a wearable device, such as a replica of a scanner, camera, or othertype of sensing device worn, for example on a lanyard. Base station102/202 is configured to determine the position and orientation ofportable user device 130/230 in real-world environment 140. As shown inFIG. 2 , in some implementations, portable user device 130/230 may be asmart device, such as a smartphone, tablet computer, or any of theexemplary replica devices described above that include smart technologysimilar to that included in a smartphone or tablet computer.

Transceiver 1161216 a and/or transceiver 216 b may be implemented aswireless communication hardware and software enabling portable userdevice 130/230 to exchange data with base station 102/202 via wirelesscommunication link 146/246. For example, transceiver 116/216 a and/ortransceiver 216 b may be implemented as fourth generation of broadbandcellular technology (4G) wireless transceivers, or as 5G wirelesstransceivers configured to satisfy the IMT-2020 requirements establishedby the International Telecommunication Union (ITU). Alternatively, or inaddition, transceiver 116/216 a and/or transceiver 216 b may beconfigured to communicate via one or more of WiFi, Bluetooth, ZigBee,and 60 GHz wireless communications methods.

Camera(s) 250 may include one or more red-green-blue (RGB) still imagecameras and/or video cameras. Moreover, in some implementations,camera(s) 250 may correspond to an array of RGB still image and/or videocameras configured to generate a panoramic image of a venue, such asreal-world environment 140.

Display 232 may take the form of a display screen, such as a touchscreendisplay implemented as a liquid crystal display (LCD), an LED display,an OLED display, or using any other suitable display technology thatperforms a physical transformation of signals to light.

P/L sensor(s) 238 may include one or more accelerometers, and/orgyroscopes, and/or a GPS receiver, and/or a magnetometer, for example.In some implementations, P/L sensor(s) 238 may be implemented as aninertial measurement unit (IMU), as known in the art.

With respect to virtual effects rendering software code 220 b, it isnoted that in some implementations, virtual effects rendering softwarecode 220 b may be a thin client application of virtual effects renderingsoftware code 120/220 a. In those implementations, virtual effectsrendering software code 220 b may enable portable user device 130/230 toprovide tracking data 224 to base station 102/202 for processing, and toreceive display data 228 including images suitable for rendering ondisplay 232, such as a video game interface or utility charts, tables,or specifications, for example. Moreover, in some implementations,virtual effects rendering software code 220 b, executed by hardwareprocessor 234 of portable use device 130/230, may detect actuation ofportable user device 130/230 by user 144, and may communicate thatdetected actuation to base station 102/202.

According to the exemplary implementation shown in FIG. 2 , virtualeffects rendering software code 220 b is located in memory 236,subsequent to transfer of virtual effects rendering software code 220 bto portable user device 130/230 over a packet-switched network, such asthe Internet, for example. Once present on portable user device 130/230,virtual effects rendering software code 220 b may be persistently storedin memory 236 and may be executed locally on portable user device130/230 by hardware processor 234.

The functionality of virtual effects rendering software code 120/220 aof base station 102/202 will be further described by reference to FIG. 3in combination with FIGS. 1A, 1B, and 2. FIG. 3 shows flowchart 360presenting an exemplary method for use by an enhanced vision system.With respect to the method outlined in FIG. 3 , it is noted that certaindetails and features have been left out of flowchart 360 in order not toobscure the discussion of the inventive features in the presentapplication.

Referring to FIG. 3 in combination with FIGS. 1A, 1B, and 2 , flowchart360 begins with detecting the presence of portable user device 130/230in real-world environment 140 (action 361). As noted above, enhancedvision system 100A/100B may include tracking device 122 communicativelycoupled to base station 102/202. Tracking device 122 may be configuredto detect the presence of portable user device 130/230 within real-worldenvironment 140, as well as to determine the position and orientation ofportable user device 130/230 in real-world environment 140. For example,and as also noted above, portable user device 130/230 may include one ormore IR emitters detectable by tracking device 122 in the form of an IRcamera or other IR sensor. Alternatively, or in addition, trackingdevice 122 may include a camera, camera array, or one or more other typeof optical sensor for detecting the presence of portable user device130/230 in real-world environment 140. Detection of the presence ofportable user device 130/230 in real-world environment 140 usingtracking device 122 may be performed by virtual effects software code120/220 a of base station 102/202, executed by hardware processor 104.

Alternatively, or in addition, in some implementations, portable userdevice 130/230 may include RFID reader 252 and/or P/L sensor(s) 238, andmay be configured to report its presence in real-world environment 140to base station 102/202. In those implementations, portable user device1301230 may utilize transceiver 216 b and wireless communication link146/246 to self-report its presence in real-world environment 140. Inthose implementations, virtual effects software code 120/220 a of basestation 102/202, executed by hardware processor 104/204, may utilizetransceiver 116/216 a and wireless communication link 146/246 to detectthe presence of portable user device in real-world environment 140 byreceiving tracking data 224 from portable user device 130/230.

Flowchart 360 continues with obtaining a mapping of real-worldenvironment 140 (action 362). In some implementations, one or morereal-world environmental maps 114/214 of real-world environment 140 maybe stored in enhanced vision database 110/210. For example, wherereal-world environment 140 is a residential venue or a commercial orindustrial venue, one or more real-world environmental maps 114/214 ofthat venue may have been generated during construction of the venue andstored in enhanced vision database 110/210, for example to map thelocations of wires, pipes, and structural support features within thewalls of those venues. In use cases in which the mapping of real-worldenvironment 140 is stored in enhanced vision database 110/210, hardwareprocessor 104/204 of base station 102/202 may execute virtual effectsrendering software code 120/220 a to obtain the mapping from enhancedvision database 110/210.

As noted above, in some implementations, enhanced vision system100A/100B may include mapping device 108/208. Mapping device 108/208 mayinclude a camera, such as a three hundred and sixty degree (360°)camera, a camera array, or one or more other type of optical sensor formapping real-world environment 140. Alternatively, or in addition,mapping device 108/208 may include a Light Detection and Ranging (lidar)device for mapping real-world environment 140. Thus, in someimplementations, obtaining the mapping of real-world environment 140 maybe performed by virtual effects software code 120/220 a of base station102/202, executed by hardware processor 104/204, and using mappingdevice 108/208.

Flowchart 360 continues with identifying one or more virtual effects112/212 for display in real-world environment 140 (action 363). In usecases in which real-world environment 140 is a residential venue or acommercial or industrial venue, and where one or more real-worldenvironmental maps 114/214 of that venue may have been generated duringconstruction of the venue to map the locations of wires, pipes, andstructural support features within the walls of those venues, forexample, virtual effects 112/212 may be imagery depicting the locationof those features. Alternatively, in use cases in which real-worldenvironment 140 is used as a venue supporting interaction by user 144with a video game, virtual effects 112/212 may include images of thegaming environment, objects, or characters to be overlaid on portions ofreal-world environment 140. Action 363 may be performed by virtualeffects software code 120/220 a of base station 102/202, executed byhardware processor 104/204, and using enhanced vision database 110/210.

In some implementations, flowchart 360 may continue with optionallymonitoring the position and orientation of portable user device 130/230in real-world environment 140 (action 364). As discussed above, enhancedvision system 100A/100B may include tracking device 122 communicativelycoupled to base station 102/202. Tracking device 122 may be configuredto monitor the position and orientation of portable user device 130/230in real-world environment 140 while tracking the movement of portableuser device 130/230 in real-world environment 140. For example, and asalso discussed above, portable user device 130/230 may include one or tomore IR emitters detectable by tracking device 122 in the form of an IRcamera or other IR sensor. Alternatively, or in addition, trackingdevice 122 may include a camera, camera array, or one or more other typeof optical sensor for monitoring the position and orientation ofportable user device 130/230 in real-world environment 140.

As another alternative, or in addition to IR tracking and/or the use ofone or more cameras, tracking device 122 may include multiple componentsdistributed within real-world environment 140 and configured to performradio-signal triangulation to monitor the position and orientation ofportable user device 130/230 in real-world environment 140. As yetanother alternative, base station 102/202 may utilize optional mappingdevice 108/208 configured to utilize a SLAM technique to monitor theposition and orientation of portable user device 130/230 in real-worldenvironment 140. Thus monitoring of the position and orientation of userdevice 130/230 in real-world environment 140 using tracking device 122and/or mapping device 108/208 may be performed by virtual effectssoftware code 120/220 a of base station 102/202, executed by hardwareprocessor 104.

Alternatively, or in addition, in some implementations portable userdevice 130/230 may include RFID reader 252 and/or P/L sensor(s) 238, andmay be configured to monitor its own position and orientation inreal-world environment 140, and to report that position and orientationto base station 102/202. In those implementations, portable user device130/230 may utilize transceiver 216 b and wireless communication link146/246 to self-report its position and orientation, i.e., the location,yaw, pitch, and roll of user device 130/230 in real-world environment140. Monitoring the position and orientation of portable user device130/230 in real-world environment 140 in response to self-reporting byportable user device 130/230 may be performed by virtual effectssoftware code 120/220 a of base station 102/202, executed by hardwareprocessor 104/204, and using transceiver 116/216 a and wirelesscommunication link 146/246. It is noted that the monitoring of theposition and orientation of portable user device 130/230 in optionalaction 364 may be performed periodically, or may be performedsubstantially continuously while portable user device 130/230 is presentin real-world environment 140.

Flowchart 360 continues with detecting actuation of portable user device130/230 in real-world environment 140 (action 365). As discussed aboveby reference to FIG. 1A, base station 102/202 may be configured tocommunicate with tracking device 122 or directly with portable userdevice 130/230 to detect actuation of portable user device 1301230 byuser 144. For example, where portable user device 130/230 is a replicaof a flashlight, actuation of portable user device 130/230 may includepointing portable user device 130/230 at an object or structure withinreal-world environment 140, such as wall structure 142 for example, andpressing a button or other actuation controller on portable user device130/230 to mimic illuminating the replica flashlight. Analogously, whereportable user device 130/230 is a replica of a laser pointer, or otherpointing device, actuation of portable user device 130/230 may includepointing portable user device 130/230 at an object or structure withinreal-world environment 140, such as wall structure 142 for example, andpressing a button or other controller on portable user device 130 tomimic actuation of the replica pointing device.

In some implementations, detecting the actuation of portable user device130/230 may be performed by virtual effects software code 120/220 a ofbase station 102/202, executed by hardware processor 104, and usingtracking device 122. However, in other implementations, detecting theactuation of portable user device 130/230 may be performed by virtualeffects software code 120/220 a of base station 1021202, executed byhardware processor 104, as the result of direct communication betweenbase station 102/202 and portable user device 130/230 via wirelesscommunication link 146/246.

Flowchart 360 can conclude with controlling display device 126A/126B todisplay one or more virtual effects 112/212, identified in action 363,in real-world environment 140, based on the mapping obtained in action362, and the position and orientation of portable user device 130/230during the actuation detected in action 365 (action 366). As discussedabove, in some implementations, the method outlined by flowchart 360 mayinclude monitoring the position and orientation of portable user device130/230 in real-world environment 140 in optional action 364. In thoseimplementations, the position and orientation of portable user device130/230 during its actuation may be determined as part of thatmonitoring action. However, in implementations in which optional action364 is omitted, actions 365 and 366 may follow directly from action 363,and position and orientation of portable user device 130/230 during theactuation detected in action 365 may be performed on the fly in responseto the detected actuation.

For example, and as noted above, enhanced vision system 100A/100B mayinclude tracking device 122 communicatively coupled to base station102/202. Tracking device 122 may be configured to determine the positionand orientation of portable user device 130/230 in real-worldenvironment 140 when the actuation of portable user device 130/230 isdetected. For instance, and as also noted above, portable user device130/230 may include one or more IR emitters detectable by trackingdevice 122 in the form of an IR camera or other IR sensor.Alternatively, or in addition, tracking device 122 may include a camera,camera array, or one or more other type of optical sensor fordetermining the position and orientation of portable user device 130/230in real-world environment 140 when the actuation of portable user device130/230 is detected.

As another alternative, or in addition to IR tracking and/or the use ofone or more cameras, tracking device 122 may include multiple componentsdistributed within real-world environment 140 and configured to performradio-signal triangulation to determine the position and orientation ofportable user device 130/230 in real-world environment 140 when theactuation of portable user device 130/230 is detected. As yet anotheralternative, base station 102/202 may utilize optional mapping device108/208 configured to utilize a SLAM technique to determine the positionand orientation of portable user device 130/230 in real-worldenvironment 140 when the actuation of portable user device 130/230 isdetected. Thus determining of the position and orientation of userdevice 130/230 in real-world environment 140 using tracking device 122and/or mapping device 108/208 may be performed by virtual effectssoftware code 120/220 a of base station 102/202, executed by hardwareprocessor 104.

Alternatively, or in addition, in some implementations portable userdevice 130/230 may include RFID reader 252 and/or P/L sensor(s) 238, andmay be configured to determine its own position and orientation inreal-world environment 140, and to report that position and orientationto base station 102/202. In those implementations, portable user device130/230 may utilize transceiver 216 b and wireless communication link146/246 to self-report its position and orientation, i.e., the location,yaw, pitch, and roll of user device 130/230 in real-world environment140 when the actuation of portable user device 130/230 is detected.Determining the position and orientation of portable user device 130/230in real-world environment 140 in response to self-reporting by portableuser device 130/230 may be performed by virtual effects software code120/220 a of base station 102/202, executed by hardware processor104/204, and using transceiver 116/216 a and wireless communication link146/246.

In some implementations, as shown in FIG. 1A, display device 126A mayinclude one or more projectors, and controlling display device 126Aprojects one or more virtual effects 112/212 onto the surface of wallstructure 142 or another surface within real-world environment 140. Asnoted above, when implemented as one or more projectors, display device126A may include a stand-alone wide-field projection system, such as aspinning POV projected illumination spinning scan line, for example.Alternatively, display device 126A may be implemented as a fisheye lensprojector, or as multiple stitched projection-mapped video projectors.

According to the exemplary implementation shown in FIG. 1B, by contrast,display device 126B of enhanced vision system 100B is wall structure 142itself, where wall structure 142 includes multiple display elements 148.According to the exemplary implementation shown in FIG. 1B, base station102/202 controls display device 126B using display data 128 to actuateat least some of display elements 148 of wall structure 142, to renderone or more virtual effects 112/212. In various implementations, and asalso noted above, wall structure 142 may be an LED wall includingdisplay elements 148 in the form of LEDs, or an organic OLED wallincluding display elements 148 in the form of OLEDs. Action 366 may beperformed by virtual effects rendering software code 120/220 a of basestation 102/202, executed by hardware processor 104/204.

In some implementations, action 366 may include determining a viewingperspective of user 144 of portable user device 1301230, and controllingdisplay device 126A/126B to display one or more virtual effects 112/212in real-world environment 140 further using the determined viewingperspective of user 144. Determining the viewing perspective of user 144may include approximating the viewing distortion experienced by user 144relative to the position and orientation of portable user device 130/230during its actuation. In some implementations, as discussed above,tracking device 122 may be configured to perform eye tracking orskeleton tracking of user 144 in real-world environment 140.Alternatively, or in addition, tracking data 124 generated by trackingdevice 122 may include optical data enabling virtual effects renderingsoftware code 120/220 a, executed by hardware processor 104/204 of basestation 102/202, to estimate the viewing perspective of user 144 basedon the distance separating the eyes of user 144 from portable userdevice 130/230, and/or based on the head position of user 144.

Referring now to FIG. 4A, FIG. 4A shows an exemplary utility applicationfor an enhanced vision system. According to the exemplary implementationshown in FIG. 4A, the enhanced vision system includes base station 402having integrated display device 426 in the form of a projection device,and portable user device 430 carried by user 444. As shown in FIG. 4A,user 444 utilizes portable user device 430 to cause virtual effects 412Ato be displayed on the surface of wall structure 442 of real-worldutility environment 440A.

Real-world utility environment 440A, wall structure 442, display device426, and user 444 correspond respectively in general to real-worldenvironment 140, wall structure 142, display device 126A, and user 144,in FIG. 1A, and those corresponding features may share any of thecharacteristics attributed to either of the respectively correspondingfeatures by the present disclosure. In addition, base station 402,portable user device 430, and virtual effects 412A correspondrespectively in general to base station 102/202, portable user device130/230, and virtual effects 112/212, in FIGS. 1A and 2 . Consequently,base station 402, portable user device 430, and virtual effects 412A mayshare any of the characteristics attributed to respective base station102/202, portable user device 130/230, and virtual effects 112/212 bythe present disclosure, and vice versa.

According to the exemplary residential or industrial utility use casedepicted in FIG. 4A, virtual effects 412A showing the locations andlayouts of pipes, electrical wiring, and wall studs, for example, can beutilized to advantage by user 144 in a number of different ways. Forinstance, virtual effects 412A may be used to mitigate damage toreal-world utility environment 440A due to a pipe leak by enabling rapididentification of the likely source of the leak. Alternatively, or inaddition, virtual effects 412A may be used to enhance the safety of user144 during repair or renovation work performed in real-world utilityenvironment 440A by identifying the locations and layouts of pipes andlive electrical wiring. In addition, or alternatively, virtual effects412A may be used to make a project such as hanging a heavy mirror orflat screen TV on wall structure 442 faster and safer by identifying thelocations of wall studs for securely anchoring those objects to wallstructure 442.

FIG. 4B shows an exemplary game application for an enhanced visionsystem. It is noted that any feature in FIG. 4B identified by areference number identical to a reference number appearing in FIG. 4Acorresponds to that previously described feature and may share any ofthe characteristics attributed to it above. As shown in FIG. 4B,real-world game environment 440B includes virtual effects 412B as animage of a game figure, such as a virtual character, projected onto asurface of wall structure 442.

In the exemplary applications shown by each of FIGS. 4A and 4B,projection of virtual effects 412A/412B by projection device 426 isperformed using a programmatic assumption of the location of user 444,based on the location of portable user device 430, to create aneye-point specific projection. The projection of virtual effects412A/412B appears to user 444 to be produced by portable user device430, although in reality, virtual effects 412A/412B are actually comingfrom projection device 426 (e.g., an overhead wide-angle projectiondevice), which may cover a large portion of real-world environments440A/440B, up to 360°.

It is noted that although FIGS. 4A and 4B depict display of virtualeffects 412A/412B using projection device 426, in other implementations,wall structure 442 may include display elements 148, shown in FIG. 1B,which may be LEDs or OLEDs, for example. In implementations in whichwall structure 442 includes display elements 148, base station 402 maybe configured to control display elements 148, rather than projectiondevice 426, to render virtual effects 412A/412B.

It is further noted that, in addition to the utility and gameapplications depicted in FIGS. 4A and 4B, the enhanced vision systemsdisclosed in the present application may have myriad other practicaluses. For instance, such an enhanced vision system can be used tovirtually redecorate a residence by generating virtual effects in theform of paint colors, wall coverings, flooring, furniture, and/or artobjects, advantageously saving a home owner the time, effort, andexpense of experimenting with real-world versions of those features. Asanother example, and by analogy, the expense and labor involved inretrofitting or otherwise upgrading a manufacturing facility may begreatly reduced utilizing the virtual effects produced by an enhancedvision system to simulate the planned facility changes.

Thus, the present application discloses enhanced vision systems andmethods that overcome the drawbacks and deficiencies in the conventionalart. According to the present novel and inventive concepts, a systemuser can experience the enhanced vision made possible by AR without thediscomfort or inconvenience associated with wearing AR eyewear orheadgear. Instead, the user may actuate a portable device, such as ahandheld device simulating a laser pointer or flashlight, for example,to point at an object or a region of the user's real-world environment.Based on the particular application of the enhanced vision system, amapping of the environment in which the portable user device isutilized, the location and orientation of the portable user device whenit is actuated, and in some implementations the perspective of the userwith respect to the object or region pointed to, virtual effectsproviding an enhanced vision experience are displayed to the user. Theenhanced vision solution disclosed by the present application mayadvantageously be utilized in a wide variety of use cases including ARgames, architectural inspection, and industrial and residential safety.

From the above description it is manifest that various techniques can beused for implementing the concepts described in the present applicationwithout departing from the scope of those concepts. Moreover, while theconcepts have been described with specific reference to certainimplementations, a person of ordinary skill in the art would recognizethat changes can be made in form and detail without departing from thescope of those concepts. As such, the described implementations are tobe considered in all respects as illustrative and not restrictive. Itshould also be understood that the present application is not limited tothe particular implementations described herein, but manyrearrangements, modifications, and substitutions are possible withoutdeparting from the scope of the present disclosure.

What is claimed is:
 1. An enhanced vision system comprising: a basestation including a hardware processor and a memory storing a virtualeffects rendering software code; a display device situated in areal-world environment having one or more surfaces and communicativelycoupled to the base station; and a portable user device separate fromthe base station and the display device; wherein the base station isconfigured to track the portable user device; and wherein the hardwareprocessor is configured to execute the virtual effects renderingsoftware code to: detect the portable user device in the real-worldenvironment; obtain a mapping of the real-world environment; identifyone or more virtual effects for display by the display device on the oneor more surfaces in the real-world environment; detect, based ontracking the portable user device in the real-world environment, aposition and an orientation of the portable user device in thereal-world environment indicative of the portable user device beingpointed at a first surface of the one or more surfaces in the real-worldenvironment; and control, based on detecting, the display device todisplay the one or more virtual effects on the first surface in thereal-world environment based on the mapping, and the position and theorientation of the portable user device.
 2. The enhanced vision systemof claim 1, wherein the hardware processor is further configured toexecute the virtual effects rendering software code to: determine aviewing perspective of a user of the portable user device; and controlthe display device to display the one or more virtual effects on thefirst surface in the real-world environment further based on thedetermined viewing perspective of the user of the portable user device.3. The enhanced vision system of claim 1, wherein the display devicecomprises a projector, and wherein the hardware processor is furtherconfigured to execute the virtual effects rendering software code tocontrol the display device to project the one or more virtual effects onthe first surface in the real-world environment.
 4. The enhanced visionsystem of claim 3, wherein the first surface is a surface of a wall, andwherein the features include wires, pipes or structural supports.
 5. Theenhanced vision system of claim 1, wherein the display device comprisesthe first surface in the real-world environment, the first surfaceincluding a plurality of display elements, and wherein the hardwareprocessor is further configured to execute the virtual effects renderingsoftware code to control the display device to activate at least one ofthe plurality of display elements to render the one or more virtualeffects.
 6. The enhanced vision system of claim 1, wherein the basestation and the portable user device are configured to be in wirelesscommunication, and wherein the base station is configured to receive theposition and the orientation of the portable user device in thereal-world environment from the portable user device.
 7. The enhancedvision system of claim 1, wherein the portable user device is one of ahandheld device or a wearable device.
 8. The enhanced vision system ofclaim 1, further comprising a tracking device communicatively coupled tothe base station, wherein the hardware processor is configured toexecute the virtual effects rendering software code to: determine, usingthe tracking device, the position and the orientation of the portableuser device in the real-world environment.
 9. The enhanced vision systemof claim 1, further comprising a mapping device communicatively coupledto the base station, wherein the hardware processor is configured toexecute the virtual effects rendering software code to: obtain, usingthe mapping device, the mapping of the real-world environment.
 10. Theenhanced vision system of claim 9, wherein the mapping device comprisesat least one of a camera or a Light Detection and Ranging (lidar)device.
 11. A method for use by an enhanced vision system including abase station having a hardware processor and a memory storing a virtualeffects rendering software code, a display device situated in areal-world environment having one or more surfaces and communicativelycoupled to the base station, and a portable user device separate fromthe base station and the display device, the base station beingconfigured to track the portable user device, the method comprising:detecting, by the virtual effects rendering software code executed bythe hardware processor, the portable user device in the real-worldenvironment; obtaining, by the virtual effects rendering software codeexecuted by the hardware processor, a mapping of the real-worldenvironment; identifying, by the virtual effects rendering software codeexecuted by the hardware processor, one or more virtual effects fordisplay by the display device on the one or more surfaces in thereal-world environment; detecting, by the virtual effects renderingsoftware code executed by the hardware processor and based on trackingthe portable user device in the real-world environment, a position andan orientation of the portable user device in the real-world environmentindicative of the portable user device being pointed at a first surfaceof the one or more surfaces in the real-world environment; andcontrolling, by the virtual effects rendering software code executed bythe hardware processor and based on detecting, the display device todisplay the one or more virtual effects on the first surface in thereal-world environment based on the mapping, and the position and theorientation of the portable user device.
 12. The method of claim 11,further comprising: determining, by the virtual effects renderingsoftware code executed by the hardware processor, a viewing perspectiveof a user of the portable user device; and wherein controlling thedisplay device to display the one or more virtual effects on the firstsurface in the real-world environment is performed further based on thedetermined viewing perspective of the user of the portable user device.13. The method of claim 11, wherein the display device comprises aprojector, and wherein controlling the display device includesprojecting the one or more virtual effects on the first surface in thereal-world environment.
 14. The method of claim 13, wherein the firstsurface is a surface of a wall, and wherein the features include wires,pipes or structural supports.
 15. The method of claim 11, wherein thedisplay device comprises the first surface in the real-worldenvironment, the first surface including a plurality of displayelements, and wherein controlling the display device includes activatingat least one of the plurality of display elements to render the one ormore virtual effects.
 16. The method of claim 11, wherein the basestation and the portable user device are in wireless communication, andwherein the base station receives the position and the orientation ofthe portable user device in the real-world environment from the portableuser device.
 17. The method of claim 11, wherein the portable userdevice is one of a handheld device or a wearable device.
 18. The methodof claim 11, wherein the position and the orientation of the portableuser device in the real-world environment is determined using a trackingdevice communicatively coupled to the base station.
 19. The method ofclaim 11, wherein obtaining the mapping of the real-world environment isperformed using a mapping device communicatively coupled to the basestation.
 20. The method of claim 19, wherein the mapping devicecomprises at least one of a camera or a Light Detection and Ranging(lidar) device.